Learning Agent for a Heat-Pump Thermostat With a Set-Back Strategy Using Model-Free Reinforcement Learning

The conventional control paradigm for a heat pump with a less efficient auxiliary heating element is to keep its temperature set point constant during the day. This constant temperature set point ensures that the heat pump operates in its more efficient heat-pump mode and minimizes the risk of activating the less efficient auxiliary heating element. As an alternative to a constant set-point strategy, this paper proposes a learning agent for a thermostat with a set-back strategy. This set-back strategy relaxes the set-point temperature during convenient moments, e.g. when the occupants are not at home. Finding an optimal set-back strategy requires solving a sequential decision-making process under uncertainty, which presents two challenges. A first challenge is that for most residential buildings a description of the thermal characteristics of the building is unavailable and challenging to obtain. A second challenge is that the relevant information on the state, i.e. the building envelope, cannot be measured by the learning agent. In order to overcome these two challenges, our paper proposes an auto-encoder coupled with a batch reinforcement learning technique. The proposed approach is validated for two building types with different thermal characteristics for heating in the winter and cooling in the summer. The simulation results indicate that the proposed learning agent can reduce the energy consumption by 4-9% during 100 winter days and by 9-11% during 80 summer days compared to the conventional constant set-point strategyComment: Submitted to Energies - MDPI.co

Demand Response of Clusters of Residential Appliances using Reduced-order Models: Bridging the Gap between Theory and Deployment

Large-scale deployment of renewable energy sources is seen as a main contributor towards the transition to a more secure, affordable and decarbonised energy system. These intermittent and unpredictable energy sources however lead to variable and difficult to predict bidirectional power flows in distribution and transmission grids. Simultaneously, the total electricity consumption is increasing as opportunities for electrification of transport and heating arise. These trends are leading to increasingly complex supply and demand power flows, pressurising the electricity grids, markets and actors that are currently in use. In recent years demand response has received renewed research attention due to advances in control theory and the abundance of affordable computational and communication resources. Harnessing the flexibility on the residential demand side to help maintain the delicate balance between production and consumption does however pose several challenges. First, the vast number of communicating devices requires a cost-competitive solution. Secondly, the development of an efficient, practical, and reliable control scheme poses a significant challenge due to the large dimensionality of the problem. Next, the comfort of the end-user should always prevail. Also, the future smart electricity grid will consist of many stakeholders, each with their specific and opposing objectives. The final challenge consists of evaluating the most suitable control methodologies, demonstrating their usefulness in simulated and physical environments. This dissertation is focussed on the development of efficient, practical, and reliable control approaches for large clusters of residential appliances. The resulting virtual power plant can be used to offer valuable services to utilities and system operators, transforming end-users into green, active and profitable stakeholders of the energy transition. To this end, an existing market-based control approach has been predominantly adopted, extended and modified. Its inherent drawbacks of imperfect state information due to aggregation, dispatch deviations resulting from the bid functions, and applicability to thermal electrical appliances have been mitigated or resolved. Throughout this work, the economic and technical feasibility of the approaches was strongly kept in mind, and supported by practical, real-world deployments. The first part of this work uses a heterogeneous mix of device classes as used in the LINEAR project, a large-scale research and demonstration project using flexible appliances of 240 Belgian families. In order to assess the effectiveness of the developed approaches, a portfolio of demand response algorithms has been implemented in a simulation framework. The integrated load-flow solver allows to evaluate the economic and technical impact of realistic scenarios. First, a novel distributed voltage control mechanism is presented that has also been deployed in the pilot. Next, this mechanism is combined with an adaptation of the market-based approach towards appliances used in the pilot. Simulation results show that from an economic welfare perspective, multiple stakeholder objectives should be simultaneously taken into account. Following this, the double-layered control approach is extended with a self-adaptive machine learning technique. By incorporating the non-linear cluster dynamics and local grid congestions, the energy cost can be further reduced. Finally, in order to bridge the gap between highly realistic simulations and a hard-wired roll-out, a deployment architecture has been developed. This allows to rapidly evaluate framework code on a physical laboratory environment, real-time digital simulator, and potentially also pilot projects. The second part is dedicated to thermostatically controlled loads, which offer the largest combined residential power and energy storage capacity. First, the market-based approach is modified towards control of heterogeneous clusters of these appliances. The novel aggregate model based on virtual tracer devices is identified in a non-intrusive manner, and captures both steady-state and transient population dynamics, as well as the heterogeneity of the cluster. Additionally, the dispatch mechanism is included in the optimization, further improving the tracking performance. Finally, a hard-wired field trial has been conducted using 25 student refrigerators in dorm rooms on the UC Berkeley campus. The developed local control modules, communication platform and control methodology simultaneously address the aforementioned barriers.1 Introduction 1.1 Smart Grids 1.2 Demand Response 1.3 Flexible Appliances 1.4 Challenges of Residential Demand Response 1.5 Research Question and Objectives 1.6 Scope of the Work, Outline and Main Contributions 2 Demand Response Simulator 2.1 Agent Based Structure 2.2 Demand Response Algorithms 2.3 Demand Response Programs 2.4 Data Models 2.4.1 Wet Appliances 2.4.2 Electric Vehicles 2.4.3 Electric Water Heaters 2.4.4 Household Load and PV Profiles 2.5 Load Flow Calculations 2.5.1 Grid Topologies 2.6 Communication Effects 2.7 Conclusion 3 Local Voltage Control using Mixed Device Classes 3.1 Related Work 3.2 Pilot Architecture 3.3 Droop Mechanism 3.4 Simulation Scenario 3.5 Simulation Results 3.6 Conclusions 4 Cluster Control of Mixed Device Classes 4.1 Three Step Approach 4.1.1 Step 1: Aggregation 4.1.2 Step 2: Optimization 4.1.3 Step 3: Real-time Control 4.2 Double Layered Control 4.3 Simulation Scenario 4.4 Simulation Results 4.5 Conclusions 5 Data-driven Control: Accounting for Modelling Inaccuracies and Grid Congestions 5.1 Markov Decision Processes 5.2 Problem Formulation 5.3 Model-based Reinforcement Learning 5.4 Evaluation 5.5 Conclusions 6 Rapid Algorithm Deployment Platform 6.1 Introduction 6.2 Algorithm Deployment Options 6.3 Architecture Overview 6.4 Hardware-in-the-loop Scenario 6.5 Simulation Results and Measurements 6.6 Conclusions 7 Cluster Control of Thermostatically Controlled Loads 7.1 Introduction 7.2 Related Work 7.3 Three Step Approach 7.3.1 Step 1: Aggregation 7.3.2 Step 2: Optimization 7.3.3 Step 3: Real-time Control 7.4 Three Step Approach for TCLs 7.4.1 Step 1: Aggregation 7.4.2 Step 2: Optimization 7.4.3 Step 3: Real-time Control 7.5 Resultsnrpages: 166status: publishe

Practical Comparison of Aggregate Control Algorithms for Demand Response with Residential Thermostatically Controlled Loads

peer reviewedThe fast deployment of renewable energy sources requires an increasing amount of frequency regulation reserves in the electric power system. Residential thermostatically controlled loads are ideal candidates to provide such services due to their thermal inertia, large energy buffers and high power ratings. However, the coordinated control of a large number of appliances represents a large-scale control problem with high computational requirements. To alleviate these needs, several aggregate modelling and control strategies have been developed in the recent literature. This paper compares two aggregate models for frequency regulation with electric water heaters used for domestic hot water: a generalized battery model and a set of representative tracer devices obtained with the cross-entropy method. Under realistic simulation assumptions, it is shown that, whereas the generalized battery model better describes the behavior of a group of electric water heaters, the tracers provide a better prediction of the available flexibility for reserves provision

Lessons learnt from the Linear large-scale energy monitoring field test

After selecting representative customers for the Linear Smart Grid project, selecting, preprocessing and transforming the gathered data according to the KDD process is a critical factor in drawing conclusions from the large-scale field test. System flaws, connection failures and unexpected human behavior constitute major challenges towards utilizing this data in a simulator environment. This paper addresses these challenges by developing a strategy for handling missing data fields and accounting for noise, outliers, and corrupted measurements. This strategy is applied to the measurement data from 69 households, which have been monitored for a one-year period with a time resolution of 15 minutes.status: publishe

Comparative analysis of aggregate battery models to characterize the flexibility of electric water heaters

peer reviewedDue to the fast deployment of renewable energy sources in the power system, an increasing amount of flexibility is required. Residential thermostatically controlled loads, such as electric water heaters, are good candidates to provide this flexibility at the demand side. However, the provision of flexibility with such appliances involves a large-scale control problem which poses computational challenges. To address these issues, a recent line of research has been dedicated to the development of aggregate models, which allow to represent a large group of thermal appliances as one entity. Within these aggregate representations, virtual battery models are particularly useful, as they provide a clear view of the available flexibility and can easily be embedded in model predictive control frameworks. In this paper, relevant quality metrics for virtual battery models are first identified. Based on a realistic group of electric water heaters, these metrics are then used to compare a series of virtual battery models developed in recent literature. The strength and weaknesses of the studied models are identified. The strong trade-off between the degree of conservativeness of the models and the reliability of their associated control actions is highlighted and quantified. It is shown that the virtual battery model and the appliances coordination strategy should be designed jointly to produce a reliable control approach. The findings of this work can be used as a starting point for the design of new virtual battery modelling approaches

Regulatory Framework for Residential Aggregators: Solutions for Low-Resolution Metering

peer reviewedIndependent aggregators and the flexibility that they unlock allow to accommodate growing shares of renewable energy sources in the power system in an affordable way. Therefore, different regulatory models have been developed and implemented to integrate these actors efficiently in electricity markets. However, these models have mostly been designed with the industry and commercial sectors in mind, and entail important barriers for aggregators when applied to residential appliances. In this paper, we propose alternative solutions to allow residential aggregators to enter markets in the short term, in cases where uncorrected operations is not allowed. If the flexible consumers have meters with monthly or yearly reading, which are still dominant in Europe, we show that imbalance perimeters can be corrected with the flexibility volumes in a simple way, without the involvement of many parties. Moreover, we argue that, due to the nature of residential flexibility, supplier compensation could reasonably be omitted. We apply these solutions in a simulation study in which electric water heaters deliver secondary frequency reserves. The arguments and results presented show that the proposed methods can be implemented to accelerate the large-scale valorization of residential flexibility

Chance-Constrained Frequency Containment Reserves Scheduling with Electric Water Heaters

peer reviewedResidential thermostatically controlled loads are good candidates to provide ancillary services to the power grid. However, some difficulties need to be overcome to harvest their flexibility. One of the challenges involved is to ensure that all controlled loads stay within their operational limits while being subject to uncertainties. In this paper, we examine the provision of frequency containment reserves with electric water heaters. We use an aggregate-and-dispatch method along with a chance-constrained optimization problem to manage uncertain frequency deviations. We reformulate the problem both in an analytical way and based on robust optimization. The approaches are then validated and compared in a simulation study. Compared to the analytical approach, we find that the robust optimization approach results in a similar average energy consumption, while considerably increasing the bidding potential of the cluster of appliances considered